Sterilization process management

ABSTRACT

Aspects of the present disclosure relate to a system including an image capture device and a computing device configured to receive a test image from the image capture device corresponding to a first surgical instrument, determine an identity type of the first surgical instrument using the test image in a machine vision technique, determine whether the first surgical instrument is flagged, and perform at least one operation in response to whether the first surgical instrument is flagged.

BACKGROUND

Surgical device reprocessing can be performed by the central sterileservices department of a hospital. Further, surgical device reprocessingcan be prone to failure due to inadequate tracking of complicated,complex processes. Any failure can result in a variety of issues for ahospital such as a cost in loss of productivity, mismanagement ofequipment and materials, and even potential harm to patients throughhealthcare associated infection.

Collecting and analyzing data regarding surgical instruments and theirmovement throughout the hospitals (from Central Sterile to the OperatingRoom and back) is important, both for managing inventory and identifyinglocation of the inventory, as well as identifying which instruments areused on which patients.

SUMMARY

While machine vision techniques have been used to identify individualsurgical instruments, machine vision techniques may have difficultylarge data sets where a large number of potential surgical instrumentsmay require significant computing resources. Further, machine visiontechniques may have issues with confidence values and lead to somequestionable classifications. Further, various processes ofsterilization management are not necessarily linked to a machine visionsystem.

Aspects of the present disclosure relate to a system including an imagecapture device and a computing device configured to receive a test imagefrom the image capture device corresponding to a first surgicalinstrument, determine an identity type of the first surgical instrumentusing the test image in a machine vision technique, determine whetherthe first surgical instrument is flagged, and perform at least oneoperation in response to whether the first surgical instrument isflagged.

Aspects of the present disclosure also relate to a system including animage capture device, and an analytical device. A computing device canbe communicatively coupled to the image capture device and theanalytical device and configured to determine, based on a machine visiontechnique from the image capture device, that a wrapped package isassembled with a group of one or more surgical instruments, receive asterilization status for the group from the analytical device, andperform at least one operation based on the sterilization status.

Aspects of the present disclosure also relate to a system including animage capture device, a surgical instrument, and a display. Thecomputing device can be communicatively coupled to the image capturedevice and the display and be configured to receive a template for aplurality of surgical instruments, the template comprises an imagedescribing position information for the plurality of surgicalinstruments, receive, from the image capture device, a video feed of thesurgical instrument, determine an instrument type and position of thesurgical instrument from the video feed, determine whether the type andposition of the surgical instrument corresponds to a surgical instrumentof the template, and perform at least one operation in response to thedetermination of the instrument type of the surgical instrument.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a block diagram of a system for performing aspects ofthe present disclosure.

FIG. 2 illustrates a block diagram of a computing device useful inperforming aspects of the present disclosure.

FIG. 3 illustrates a flowchart of a method of performing an operation inresponse to a flagged event, according to various embodiments.

FIG. 4 illustrates a flowchart of a method of determining sterility of awrapped package, according to various embodiments.

FIG. 5 illustrates a flowchart of a method of managing identification ofa surgical instrument, according to various embodiments.

FIG. 6 illustrates a continuation of the method of FIG. 5, according tovarious embodiments.

FIG. 7 illustrates a flowchart of a method of assembling a load assistdevice, according to various embodiments.

FIG. 8 illustrates a flowchart of a method of generating a template,according to various embodiments.

FIG. 9 illustrates an image of a user interface featuring a template,according to various embodiments.

FIG. 10 illustrates an image of a user interface featuring a templatewith zoom capability, according to various embodiments.

FIG. 11 illustrates an image of a user interface featuring a templatewith zoom capability and view selection, according to variousembodiments.

FIG. 12 illustrates an image of a user interface featuring a templatewith a chemical indicator, according to various embodiments.

FIG. 13 illustrates a block diagram of a training process, according tovarious embodiments.

FIG. 14 illustrates a flowchart of a method of determining a trainingevent, according to various embodiments.

FIG. 15 illustrates a flowchart of a method of determining a trainingevent, according to various embodiments.

FIG. 16 illustrates a flowchart of a method of determining a trainingevent, according to various embodiments.

FIG. 17 illustrates a user interface featuring a technician managementinterface, according to various embodiments.

FIG. 18 illustrates a block diagram of a datastore management process,according to various embodiments.

FIG. 19 illustrates an image of an article message using bar codestrips, according to various embodiments.

DETAILED DESCRIPTION

Aspects of the present invention can provide operations in response toflagged events and be useful in combining sterilization management withmachine vision techniques.

FIG. 1 is a block diagram illustrating an example system 100 foroptically determining an identity type of a surgical instrument 112. Asshown in FIG. 1, system 100 includes an image capture device 102. Imagecapture device 102 may include one or more image capture sensors 106 andone or more light sources 104.

The one or more surgical instruments 112 can be placed in a load assistdevice 113 (which can withstand a sterilization process). In at leastone embodiment, the load assist device 113 can be a wheeled cart capableof receiving multiple surgical instruments 112. The load assist device113 can also be a tray capable of receiving multiple surgicalinstruments 112. The load assist device 113 can be configured to beplaced in a sterilizer. To facilitate recognizing the borders of theload assist device 113, one or more optically active articles can beplaced around the perimeter of a load assist device 113 (e.g., aroundthe top perimeter of a tray) The optically active article can includeretroreflective, or reflective tape such that a vision-based system canrecognize the border of the load assist device 113.

In at least one embodiment, a load assist device 113 or the surgicalinstrument 112 can be placed on a background surface 128. The backgroundsurface 128 is a surface that can facilitate determining an identitytype of the surgical instrument 112. The background surface 128 can alsobe optically contrasting with the surgical instrument (e.g., a bluenonwoven for a stainless-steel instrument 112).

The background surface 128 can have markers embedded or printed thereonwhich facilitates scaling of the image. For example, hatch-marks orgrids can be formed on the background surface to provide scale for thesurgical instrument 112 and may be used to gauge the relative sizes ofthe instruments. In some embodiments, the scale can comprise a rulerthat is found within the pack of instruments.

The background surface 128 can also include regions responsive tocertain instrument types. For example, the background surface 128 canhave a region for locking instruments, clamping instruments, a regionfor lumened instruments, etc. to facilitate identification of theinstrument by the system 100. Such a regioned area can specificallyfacilitate supervised learning (as described herein. The backgroundsurface 128 can interface with the UI Component 120 to provide controlsfor the computing device 116. For example, the background surface 128can have tactile buttons, projected interface buttons, gestural touch,etc.

In at least one embodiment, the background surface 128 can include ascale 136 to measure a mass of the surgical instrument. The mass of thesurgical instrument can be identified as a feature and combined with theimage analysis to aid in identifying the identity type of the surgicalinstrument. For example, if there is an 88% confidence that an image ofa surgical instrument is an alligator forceps, and the mass measuredindicates that the surgical instrument is approximately the same mass asalligator forceps, then the confidence can be increased by the computingdevice 116.

Further, the background surface can also include markers which canfurther facilitate outlines of the surgical instruments. For example, amicroreplicated surface in a rectilinear array to provide a greaternumber of reference points. The background surface 128 can be formed ofany material such as a nonwoven and be relatively flat to provideminimal lifting of the surgical instrument (such as a mat or a sheet).An example background surface is found on FIG. 15, which is shown withsurgical instruments disposed thereon. Both the load assist device 113and background surface 128 can be optional.

System 100 may also include one or more optically active articles 108 asdescribed in this disclosure, such as labels, tags, or plates. Theoptically active article 108 can be attached to or printed on a group ofsurgical instruments. For example, the group of surgical instruments canbe placed in a wrapped package of surgical instruments and the opticallyactive article 108 is placed on the outside of the wrapped package.

In at least one embodiment, the optically active article 108 is disposedproximate to the group of surgical instruments. The optically activearticle 108 can be associated with a group of surgical instruments in adatastore 129 (described herein). The wrapped package can be createdafter the surgical instruments are cleaned and rinsed but prior tosterilization in a sterilizer 125.

In at least one embodiment, the optically active article 108 may have anarticle message 126 that includes a QR code, bar code, text string orother coded or uncoded visual identifier from the load assist device 113or background surface 128 to learn information about the bundle. Forexample, the system may detect a barcode present inside the unwrappedbundle, conclude that all the instruments are expected to be from thesame manufacturer or from an expected (small) set of instruments. Thisinformation will advantageously reduce the search space for theidentification task.

The reflective, non-reflective, and/or retroreflective sheet may beapplied to a base surface using one or more techniques and/or materialsincluding but not limited to: mechanical bonding, thermal bonding,chemical bonding, or any other suitable technique for attachingretroreflective sheet to a base surface. A base surface may include anysurface of an object (such as described above, e.g., an aluminum plate)to which the reflective, non-reflective, and/or retroreflective sheetmay be attached. An article message may be printed, formed, or otherwiseembodied on the sheeting using any one or more of an ink, a dye, athermal transfer ribbon, a colorant, a pigment, and/or an adhesivecoated film. In some examples, content is formed from or includes amulti-layer optical film, a material including an optically activepigment or dye, or an optically active pigment or dye.

Article message in FIG. 1 is described for illustration purposes asbeing formed by different areas that either retroreflect or do notretroreflect light. An article message in FIG. 1 may be printed, formed,or otherwise embodied in an optically active article using any lightreflecting technique in which information may be determined from thearticle message. For instance, article message 126 may be printed usingvisibly-opaque, infrared-transparent ink and/or visibly-opaque,infrared-opaque ink. Any suitable construction, in which article message126 or portions thereof are distinguishable under one or more lightingconditions, may be used in accordance with techniques and articles ofthis disclosure.

In at least one embodiment, the identity type can be a class aninstrument 112 (e.g., non-locking forceps). The identity type can alsobe more specific to the subclass or variety of instrument (e.g., arteryforceps). In at least one embodiment, the identity type can be anindividual instrument (e.g., instrument number 344444).

In some examples, image capture device 102 is communicatively coupled tocomputing device 116 via network 114 using one or more communicationlinks. In other examples, as described in this disclosure, image capturedevice 102 may be communicatively coupled to computing device 116 viaone or more forms of direct communication without network 114, such asvia a wired or wireless connection that does not require a network.

Image capture device 102 may convert light or electromagnetic radiationsensed by image capture sensors 106 into information, such as digitalimage or bitmap comprising a set of pixels. Each pixel may havechrominance and/or luminance components that represent the intensityand/or color of light or electromagnetic radiation. While the imagecapture device 102 can be any shape and mount in any position, exemplaryimage capture devices 102 are shown on FIGS. 16 (ceiling mounted), 17(ceiling mounted, freestanding, or wall mounted), 18 (wall mounted).

In at least one embodiment, a capability of the imaging system torecognize objects (instruments 112) moving “into” or “out of” a givenfield within the field of vision of the image capture device 102. Forexample, if a camera is in a fixed mounted position overlooking an ORroom, and the sterile field is in a fixed position in the room, couldthe imaging system demarcate that area within its field of vision anddenote an area that's considered “in” that space or “not” in that space(e.g., defined by the area of the background surface 128). The trackingof instruments 112 “into” that field and when it went “out of” thatfield can imply its use of the instrument (e.g., in an operating roomsetting).

Image capture device 102 may include one or more image capture sensors106 and one or more light sources 104. In some examples, image capturedevice 102 may include image capture sensors 106 and light sources 104in a single integrated device. In other examples, image capture sensors106 or light sources 104 may be separate from or otherwise notintegrated in image capture device 102. In at least one embodiment, aseparate light source 104 can at least be proximate to the image capturedevice 102. Examples of image capture sensors 106 may includesemiconductor charge-coupled devices (CCD) or active pixel sensors incomplementary metal-oxide-semiconductor (CMOS) or N-typemetal-oxide-semiconductor (NMOS, Live MOS) technologies. Digital sensorsinclude flat panel detectors. In the example, image capture device 102includes at least two different sensors for detecting light in twodifferent wavelength spectrums. In some embodiments, a first imagecapture and a second image capture sensor substantially concurrentlydetect the first and second wavelengths. Substantially concurrently mayrefer to detecting the first and second wavelengths within 10milliseconds of one another, within 50 milliseconds of one another, orwithin 100 milliseconds of one another to name only a few examples. Inat least one embodiment, the light source 104 can illuminate thesurgical instrument 112 so that uniform lighting is created.

In some examples, one or more light sources 104 include a first sourceof radiation and a second source of radiation. In some embodiments, thefirst source of radiation emits radiation in the visible spectrum, andthe second source of radiation emits radiation in the near infraredspectrum. In other embodiments, the first source of radiation and thesecond source of radiation emit radiation in the near infrared spectrum.In at least one embodiment, the first wavelength may be visible lightwhile the second wavelength can be ultraviolet light. In one example,one or more light sources 104 may emit radiation (e.g., light 127) inthe near infrared spectrum.

In some examples, image capture device 102 includes a first lens and asecond lens. In some examples, image capture device 102 captures framesat 50 frames per second (fps). Other exemplary frame capture ratesinclude 60, 30 and 25 fps. It should be apparent to a skilled artisanthat frame capture rates are dependent on application and differentrates may be used, such as, for example, 10 or 20 fps. Factors thataffect required frame rate are, for example, application (e.g., movingcart vs stationary cart), vertical field of view.

In some examples, image capture device 102 includes at least twochannels. The channels may be optical channels. The two optical channelsmay pass through one lens onto a single sensor. In some examples, imagecapture device 102 includes at least one sensor, one lens and one bandpass filter per channel. The band pass filter permits the transmissionof multiple near infrared wavelengths to be received by the singlesensor. The at least two channels may be differentiated by one of thefollowing: (a) width of band (e.g., narrowband or wideband, whereinnarrowband illumination may be any wavelength from the visible into thenear infrared); (b) different wavelengths (e.g., narrowband processingat different wavelengths can be used to enhance features of interest,while suppressing other features (e.g., other objects, sunlight,headlights); (c) wavelength region (e.g., broadband light in the visiblespectrum and used with either color or monochrome sensors); (d) sensortype or characteristics; (e) time exposure; and (f) optical components(e.g., lensing).

The image capture device 102 may be stationary or otherwise mounted in afixed position relative to the surgical instrument 112. For example, theimage capture device 102 can be mounted on a workstation, on the ceilingoverlooking an individual workstation, etc. In some examples, imagecapture device 102 may be worn (continuously) or held by a humanoperator (such as glasses, augmented reality, part of a uniform, or on alanyard) or robotic device, which changes the position of image capturedevice 102 relative to the surgical instrument 112.

Image capture device 102 may be communicatively coupled to computingdevice 116 by one or more communication links 130A and 130B. Imagecapture device 102 may send images of surgical instrument 112 tocomputing device 116.

Communication links 130A, 130B, and 130C may represent wired or wirelessconnections. For instance, communication links 130A and 130B may bewireless Ethernet connections using a WiFi protocol and/or may be wiredEthernet connections using Category 5 or Category 6 cable. Any suitablecommunication links are possible. In some examples, image capture device102 is communicatively coupled to computing device 116 by a network 114.Network 114 may represent any number of one or more network connecteddevices including by not limited to routers, switches, hubs, andinterconnecting communication links that provide for forwarding ofpacket and/or frame-based data. For instance, network 114 may representthe Internet, a service provider network, a customer network, or anyother suitable network. In other examples, image capture device 102 iscommunicatively coupled to computing device 116 by a direct connection,such as Universal Serial Bus (USB) link or other high-speed bus.Although shown separately in FIG. 1, image capture device 102 andcomputing device 116 may be integrated in a single device or housing.The single device or housing may be attached to a building or otherstationary structure, or may not be stationary such that a humanoperator may carry the single device or housing as a portable structure.

Computing device 116 represents any suitable computing system, which maybe remote from or tightly integrated with image capture device 102, suchas one or more desktop computers, laptop computers, mainframes, servers,cloud computing systems, etc. capable of sending and receivinginformation with image capture device 102. In some examples, computingdevice 116 implements techniques of this disclosure. Using techniques ofthis disclosure, computing device 116 may determine whether the group ofone or more surgical instruments enters or exits the sterilizer 125.

In the example of FIG. 1, computing device 116 includes a technicianperformance component 117 (which can implement the methods described inFIGS. 13-16), an analysis component 118 (which can perform the imagerecognition analysis), a data layer 120, service component 122 and userinterface (UI) component 124. An analysis component 118 can receive animage, convert the image into one or more feature values, and determinethe instrument type of the surgical instrument in the image. Theanalysis component 118 can further query a datastore 129 to receiveinformation related to the surgical instrument.

Further, the computing device 116 can be communicatively coupled to adatastore 129 via the network 114 or a direct connection. The datastore129 may store data in structure or unstructured form. Example datastoresmay be any one or more of a relational database management system,online analytical processing database, table, or any other suitablestructure for storing data.

The datastore 129 can have one or more records 132 which are associatedwith a group of one or more surgical instruments 112. The record 128 forsurgical instruments 112 can be accessed based on the input from theimage capture device 102. For example, after every entry or exit of thegroup of one or more surgical instruments 112 into or out of thesterilizer (which is detected by the image capture device 102identifying the surgical instrument 112), the record 132 for anyindividual surgical can be subject to a record management operation. Inat least one embodiment, comparing a surgical instrument 112 to a record132 in the database for comparison purposes will help determine thenumber of uses for that particular instrument (through how many timesthis instrument has been seen by vision system 100) and can allow forbetter maintenance planning and inventory management.

The record management operation is any operation that changes the record(e.g., creates a new attribute or record, modifies an attribute of anexisting record, or deletes an attribute). In at least one embodiment,the record management operation includes modifying a record in thedatastore for the one or more surgical instruments that the one or moresurgical instruments are present (e.g., in a package). In the exampleabove, a record for each surgical instrument in the group can be updatedto indicate that the surgical instrument was sterilized (or at leastplaced in the sterilizer) upon both check-in and check-out of the groupfrom the sterilizer.

While information related to static data is constant and non-updatable,information related to a surgical instrument may be updated. In oneinstance, where database is integrated with or part of a machine visionsystem, information related to surgical instrument (s) may be updated ona regular and recurring basis, such as downloaded through network 114daily. In one instance, information related to a surgical instrument maybe stored in a centralized datastore 129 connected to a network thatallows multiple individuals or entities to update information related tosurgical instrument. Machine vision systems or other entities requiringaccess to information related to surgical instruments can then updateinformation from the centralized datastore 129 so that it can be storedlocally and a local copy can be accessed in real time, independent of anaccessible network connection. In another instance, a machine visionsystem may connect with and communicate through network 114 to query adatabase or a centralized datastore 129 storing information related tosurgical instrument (s). In some examples, a computing device may querymultiple different databases, and in some examples, the datastore 129queried from the multiple different datastores may be based at least inpart on data included for the surgical instrument.

Information related to surgical instruments can include a wide range ofinformation. Some examples of information related to surgicalinstruments are: a condition of the article, a condition of the physicalarea near the article, identifying information for a person to whom thearticle is assigned, instructions for a user of the article, andinstructions for an individual or device in proximity to the article.

Service component 122 may provide any number of services, by performingone or more operations. For instance, service component 122, uponreceiving an instrument type may generate one or more alerts, reports,or other communications that are sent to one or more other computingdevices. Such alerts may include but are not limited to: emails, textmessages, lists, phone calls, or any other suitable communications.

In some examples, user interface (UI) component 124 may act as anintermediary between various components and optical elements ofcomputing device 116 to process and send input detected by input devicesto other components and optical elements, and generate output from othercomponents and optical elements that may be presented at one or moreoutput devices. For instance, UI component 124 may generate one or moreuser interfaces for display, which may include data and/or graphicalrepresentations of alerts, reports, or other communications.

Components 118, 120, 122, and 124 may perform operations describedherein using software, hardware, firmware, or a mixture of bothhardware, software, and firmware residing in and executing on computingdevice 116 and/or at one or more other remote computing devices. In someexamples, components 118, 120, 122, and 124 may be implemented ashardware, software, and/or a combination of hardware and software.Computing device 116 may execute components 118, 120, 122, and 124 withone or more processors.

Computing device 116 may execute any of components 118, 120, 122, and124 as or within a virtual machine executing on underlying hardware.Components 118, 120, 122, and 124 may be implemented in various ways.For example, any of components 118, 120, 122, and 124 may be implementedas a downloadable or pre-installed application or “app.” In anotherexample, any of components 118, 120, 122, 124 may be implemented as partof an operating system of computing device 116. In any case, components118, 120, 122, and 124 may execute at or be implemented at computingdevices described herein, which may be an example of computing device116.

In at least one embodiment, the datastore 129 can include a contentdatastore (not shown) that may include a series of bits consisting ofthe payload from content optical elements and the information associatedwith those series of bits. In some examples, the content datastore mayinclude messages in encoded or decoded form. The datastore 129 can alsoinclude a context datastore (not shown) which may include a series ofbits consisting of the payload from context optical elements and theinformation associated with those series of bits. In some examples, thecontext datastore may include messages in encoded or decoded form. Thedatastore 129 can also include Error Correction Data which may includeseries bits forming codewords constructed by the error correctionalgorithm which aids in reconstruction and verification of payload datafound in the content optical elements and context optical elements. Thedatastore 129 can include service data which may include any data toprovide and/or resulting from providing a service of service component.For instance, service data may include information about opticallyactive articles (e.g., sterilization check-in/out), user information, orany other information.

In at least one embodiment, an image of the surgical instrument 112 iscaptured with light in the visible light spectrum. In some examples, afirst spectral range is from about 350 nm to about 700 nm (i.e., visiblelight spectrum) and a second spectral range is from about 700 nm toabout 1 100 nm (i.e., near infrared spectrum). In some examples, a firstspectral range is from about 700 nm to about 850 nm, and a secondspectral range is between 860 nm to 100 nm. In another example, thefirst or second spectral range can be ultraviolet spectrum (10 nm to 400nm) which can also further sterilize the one or more surgicalinstruments. When an image is generated, the visible light 127 isretroreflected back to image capture device 102. As a result ofreceiving the retroreflected light, article message 126 may appearblack, while portions other than article message 126 may appear white orbright relative to. In at least one embodiment, the image capture device102 can capture only visible light.

In some examples, the first lighting condition includes a first range ofwavelengths and the second lighting condition includes a second range ofwavelengths that is substantially different from the first range ofwavelengths. In some examples, first and second ranges of wavelengthsmay be substantially different if less than 1% of the wavelengths arethe same in each range of wavelengths. In some examples, first andsecond ranges of wavelengths may be substantially different if the fewerthan between 1% and 10% of the wavelengths are the same in each range ofwavelengths. In some examples, first and second ranges of wavelengthsmay be substantially different if the number of wavelengths are the samein each range is less than a threshold amount.

In some examples, a second image may be captured under UV lighting. Foran image of surgical instrument 112 captured under UV lightingconditions, soil or other contaminants may be more visible to the imagecapture device 102.

The system 100 can also include various analytical devices 140 thatcommunicate with the computing device 116 secondary properties of thesurgical instrument 112. The properties of the surgical instrument 112can be associated with a record 132 for the surgical instrument.

The analytical devices 140 can include an indicator 134 and a scale 136.In at least one embodiment, the computing device 116 can receive astatus of the indicator 134 from the image capture device 102. Forexample, the indicator 134 can be a chemical indicator 134 that isembedded, attached to, or proximate to the optically active article 108.An example of an indicator 134 is commercially available under the tradedesignation Comply from 3M (St Paul, Minn.). The computing device 116can determine whether the chemical indicator is present with the one ormore surgical instruments based on first image or the second image anddetermine the stats of the chemical indicator 134 from the first orsecond image. In at least one embodiment, the computing device 116 canperform at least one operation in response to the determination of thepresence or status of the chemical indicator 134 (e.g., modify a recordfor the one or more surgical instruments in the group). In at least oneembodiment, the indicator portion of the is responsive to anenvironmental condition.

In at least one embodiment, the indicator 134 would be detected by thecomputing device 116 at two-time points in a workflow associated withreprocessing and use of the instruments in a given load assist device113: 1) at the time the load assist device 113 is assembled afterdecontamination of the instruments 112 and prior to being wrapped with anonwoven sheet, and 2) at the time that a nonwoven sheet containing thesurgical instrument 112 is opened in the operating room and the contentsof the tray are available for use.

In the first instance above, the vision system 100 can verify that atechnician assembling the load assist device 113 had in fact placed thechemical indicator 134 in the load assist device 113 prior to wrappingit. This would ensure that a wrapped load assist device 113 would not belater opened in an operating room only to find out that there was amissing chemical indicator 134 in that load assist device 113 whichwould require reprocessing of the surgical instrument 112. This cancreate havoc and a significant delay in the OR, especially if the traycontains specialized instrumentation.

In the second instance above, the vision system 100 would be used todetect and log the actual result (pass/fail) from that chemicalindicator 134 automatically providing updates to the record 132 for theinstrument 112. The chemical indicator 134 can use a similar detectionmethod as the instrument 112. The chemical indicator 134 shape can betreated as another shape to detect. A machine learning classifier, forexample a multiclass logistic regression classifier, would be trained todistinguish multiple surgical instruments 112 from each other, and beable to distinguish the appearance of the chemical indicator 134 fromthat of the surgical instrument 112.

In at least one embodiment, the machine vision system 100 can alsoinclude a computing device 138. The computing device 138 can also be aclient device to the computing device 116. Although one computing device138 is shown, multiple computing devices 138 can exist in the system.The computing device 138 can be a laptop, smart phone, augmented realitydevice, virtual reality device, or other device that receivesinformation and processes the information in a machine-readable format.The computing device 138 can receive templates from the datastore 129and display the video feed from the image capture device 102. In atleast one embodiment, the computing device 138 can also have a bar codescanner. The computing device 138 can receive alerts communicated fromthe computing device 116. For example, if a surgical instrument ismissing from an operating room, then the computing device 116 can alertcomputing device 138 which is carried by a technician. The techniciancan promptly retrieve the surgical instrument from one part of thehospital and deliver the instrument to the operating room.

FIG. 2 is an example of a computing device for use in a dynamic systemfor determining an instrument type. FIG. 2 illustrates only oneparticular example of computing device 116, as shown in FIG. 1. Manyother examples of computing device 116 may be used in other instancesand may include a subset of the components included in example computingdevice 116 or may include additional components not shown examplecomputing device 116 in FIG. 2. In some examples, computing device 116may be a server, tablet computing device, smartphone, wrist- orhead-worn computing device, laptop, desktop computing device, or anyother computing device that may run a set, subset, or superset offunctionality included in application 228.

As shown in the example of FIG. 2, computing device 116 may be logicallydivided into user space 202, kernel space 204, and hardware 206.Hardware 206 may include one or more hardware components that provide anoperating environment for components executing in user space 202 andkernel space 204. User space 202 and kernel space 204 may representdifferent sections or segmentations of memory, where kernel space 204provides higher privileges to processes and threads than user space 202.For instance, kernel space 204 may include operating system 220, whichoperates with higher privileges than components executing in user space202.

As shown in FIG. 2, hardware 206 includes one or more processors 208,input components 210, storage devices 212, communication units 214, andoutput components 216. Processors 208, input components 210, storagedevices 212, communication units 214, and output components 216 may eachbe interconnected by one or more communication channels 218.Communication channels 218 may interconnect each of the components 208,210, 212, 214, and 216 for inter-component communication (physically,communicatively, and/or operatively). In some examples, communicationchannels 218 may include a hardware bus, a network connection, one ormore inter-process communication data structures, or any othercomponents for communicating data between hardware and/or software.

One or more processors 208 may implement functionality and/or executeinstructions within computing device 116. For example, processors 208 oncomputing device 116 may receive and execute instructions stored bystorage devices 212 that provide the functionality of componentsincluded in kernel space 204 and user space 202. These instructionsexecuted by processors 208 may cause computing device 216 to storeand/or modify information, within storage devices 212 during programexecution. Processors 208 may execute instructions of components inkernel space 204 and user space 202 to perform one or more operations inaccordance with techniques of this disclosure. That is, componentsincluded in user space 202 and kernel space 204 may be operable byprocessors 208 to perform various functions described herein.

One or more input components 210 of computing device 116 may receiveinput. Examples of input are tactile, audio, kinetic, and optical input,to name only a few examples. Input components 242 of computing device216, in one example, include a mouse, keyboard, voice responsive system,video camera, buttons, control pad, microphone or any other type ofdevice for detecting input from a human or machine. In some examples,input component 210 may be a presence-sensitive input component, whichmay include a presence-sensitive screen, touch-sensitive screen, etc.

One or more output components 216 of computing device 116 may generateoutput. Examples of output are tactile, audio, and video output. Outputcomponents 216 of computing device 116, in some examples, include apresence-sensitive screen, sound card, video graphics adapter card,speaker, cathode ray tube (CRT) monitor, liquid crystal display (LCD),or any other type of device for generating output to a human or machine.Output components may include display components such as cathode raytube (CRT) monitor, liquid crystal display (LCD), Light-Emitting Diode(LED) or any other type of device for generating tactile, audio, and/orvisual output. Output components 216 may be integrated with computingdevice 116 in some examples. In other examples, output components 216may be physically external to and separate from computing device 116,but may be operably coupled to computing device 116 via wired orwireless communication. An output component may be a built-in componentof computing device 116 located within and physically connected to theexternal packaging of computing device 116 (e.g., a screen on a mobilephone). In another example, presence-sensitive display 202 may be anexternal component of computing device 116 located outside andphysically separated from the packaging of computing device 116 (e.g., amonitor, a projector, etc. that shares a wired and/or wireless data pathwith a tablet computer).

One or more communication units 214 of computing device 116 maycommunicate with external devices by transmitting and/or receiving data.For example, computing device 116 may use communication units 214 totransmit and/or receive radio signals on a radio network such as acellular radio network. In some examples, communication units 214 maytransmit and/or receive satellite signals on a satellite network such asa Global Positioning System (GPS) network. Examples of communicationunits 214 include a network interface card (e.g. such as an Ethernetcard), an optical transceiver, a radio frequency transceiver, a GPSreceiver, or any other type of device that can send and/or receiveinformation. Other examples of communication units 214 may includeBluetooth®, GPS, 3G, 4G, and Wi-Fi® radios found in mobile devices aswell as Universal Serial Bus (USB) controllers and the like.

One or more storage devices 212 within computing device 116 may storeinformation for processing during operation of computing device 116. Insome examples, storage device 212 is a temporary memory, meaning that aprimary purpose of storage device 212 is not long-term storage. Storagedevices 212 on computing device 116 may configured for short-termstorage of information as volatile memory and therefore not retainstored contents if deactivated. Examples of volatile memories includerandom access memories (RAM), dynamic random access memories (DRAM),static random access memories (SRAM), and other forms of volatilememories known in the art.

Storage devices 212, in some examples, also include one or morecomputer-readable storage media. Storage devices 212 may be configuredto store larger amounts of information than volatile memory. Storagedevices 212 may further be configured for long-term storage ofinformation as nonvolatile memory space and retain information afteractivate/off cycles. Examples of non-volatile memories include magnetichard discs, optical discs, floppy discs, flash memories, or forms ofelectrically programmable memories (EPROM) or electrically erasable andprogrammable (EEPROM) memories. Storage devices 212 may store programinstructions and/or data associated with components included in userspace 202 and/or kernel space 204.

As shown in FIG. 2, application 228 executes in user space 202 ofcomputing device 116. Application 228 may be logically divided intopresentation layer 222, application layer 224, and data layer 226.Presentation layer 222 may include user interface (UI) component 124,which generates and renders user interfaces of application 228.Application 228 may include, but is not limited to: UI component 124,algorithm coding component 542, data layer 226, and one or more servicecomponents 546. Presentation layer 222 may include UI component 124.

Data layer 226 may include one or more datastores (defined herein).Content datastore 234 may include: a series of bits consisting of thepayload from content optical elements and the information associatedwith those series of bits. In some examples, content datastore 234 mayinclude messages in encoded or decoded form. Context datastore 236 mayinclude a series of bits consisting of the payload from context opticalelements and the information associated with those series of bits. Insome examples, context datastore 234 may include messages in encoded ordecoded form. Error Correction Data 232 may include series bits formingcodewords constructed by the error correction algorithm which aids inreconstruction and verification of payload data found in the contentoptical elements and context optical elements. Service data 238 mayinclude any data to provide and/or resulting from providing a service ofservice component 122. For instance, service data may includeinformation about optically active articles (e.g., vehicle registrationinformation), user information, or any other information.

A machine vision system 100 may capture an image including the surgicalinstrument, determine an instrument type and communicate it to computingdevice 116 through UI component 124 or communication channels. Servicecomponent 122 may perform one or more operations based on the datagenerated by analysis component 118, such as send data to UI component124 that causes UI component 124 to generate an alert for display. Otheroperations may include generating a report or storing a message based ondata generated by an analysis component 118. In some examples, servicecomponent 122 may modify the record 132 of a central datastore 129

While one particular implementation of a computing system is describedherein, other configurations and embodiments of computing systemsconsistent with and within the scope of the present disclosure will beapparent to one of skill in the art upon reading the present disclosure.

FIG. 3 is a flowchart of a method 300 of conducting an inventory. Atleast a portion of the method can be implemented by the computing devicedescribed herein.

In block 310, the computing device can receive a test image from theimage capture device or from the datastore or a remote source. The testimage can correspond to a surgical instrument. The test image caninclude one or more images of the surgical instrument and received as avideo feed from the image capture device. In at least one embodiment,the test image can be retrieved from the datastore and be relevant wherea technician encounters an unknown instrument having an unlabeled imageand wants to determine the identity at a later time. This can beparticularly useful during training.

In block 320, the computing device can determine an identity type of thesurgical instrument using the test image in a machine vision technique.Machine vision techniques can be described in the copending provisionalapplication titled “Machine vision system for surgical instrumentidentification”, which is incorporated by reference in its entirety. Themachine vision techniques described herein can also be used to identifya chemical indicator and a status of the chemical indicator.

An embodiment of an overall inventory management method is found onFIGS. 5 and 6. In at least one embodiment, the determining of theidentity type can be conditional based on the surgical instrument notbeing previously labeled. This can save computing resources forunlabeled or untagged surgical instruments by reducing function calls tothe machine vision system.

For example, turning to FIG. 5, new items can be received for intakeinto the inventory database. If a PO or package receipt is available,that artifact can be scanned using a device that runs the image againstan OCR engine that interprets the image to parse out the informationinto individual text content that is then matched against a GlobalManufacturer database (GMDB) to find a match. If there is a match, thedetails for each line item is automatically generated and populated intothe system. If the image cannot be interpreted, the user can eitherrecapture an image to repeat the process, or the user can manuallyidentify the item(s) against the GMDB. If no PO or package receipt isavailable, the user may be able to use each item's bar code or assignedQR code to match against the GMDB. If there are no bar codes or QRcodes, the user may capture an image of the item(s) itself/themselves,which is then sent to an Image Recognition Engine (IRE) that analysesthe image to match against the GMDB. If the IRE cannot establish amatch, or the user cannot capture an image, the user can ultimatelymanually find the item in the GMDB or create a new item entry into thefacility's Local Manufacturer database (LMDB). Once all items have beenproperly identified by the system, those items are now ready to beassociated to corresponding trays as needed.

In at least one embodiment, the computing device can determine, with theimage capture device, whether an article message is present on asurgical instrument as described in process group 512, and determine theidentity type of the first surgical instrument using a machine visiontechnique based on whether the article message is present as shown inprocess group 514. As discussed herein, the article message can be a barcode. QR code, visual indication of an alphanumeric sequence, orcombinations thereof. An exemplary bar code can further comprise acolor-coded bar code stripes as shown in FIG. 19.

Returning to FIG. 5, the computing device can also determine whether thearticle message is present occurs responsive to determining if a receiptor purchase order is available for the device as shown in process group510. If available, the computing device adding the receipt or purchaseorder to the record for the surgical instrument. In at least oneembodiment, the purchase order can be used to identify the surgicalinstrument via optical character recognition.

Returning to FIG. 3, m block 330 and 340, the computing device canreceive and identity parameter which can be used to add confidence tothe identity of the instrument type. The identity parameter can be asecondary identity which helps aid the identification of the test image.For example, a scale can be used to determine mass of the surgicalinstrument which can be paired with the image analysis to increaseconfidence in the overall model.

Once determined, the computing device can adjust a confidence of thedetermination of the identity type based on an identity parameter. Forexample, item mass and refraction index can be paired with the machinevision analysis of the image to increase confidence in the assessment.

In block 350, the computing device can access a record for a surgicalinstrument. The record can be maintained in a datastore as discussedherein.

In block 360, the computing device can determine whether the firstsurgical instrument is flagged. In at least one embodiment, a flag is acondition associated with the record for the surgical instrument thatindicates to the computing device to perform an operation. The conditioncan be conditional operation derived from the one or more records fromthe datastore. In at least one embodiment, the flag is generated by thecomputing component based on attributes within the datastore. While manytypes of flags can exist, for the purposes of illustration, only a fewflags will be discussed.

For example, a flag can be based on special handling instructions forthe surgical instrument. A difficult to clean surgical instrument can beflagged based on cleanliness results from a biological, chemicalindicator, or an ATP-based indicator. The record can be flagged by thesystem automatically as a result of receiving special cleaninginstructions from a manufacturer. An operation that can be triggeredupon receiving the flag is to present the special handling instructionson the display.

A flag can also be based on an urgent request. For example, a surgicalinstrument may be checked-in but another surgical instrument of the sametype is urgently needed in an operating room. Thus, the operation can beto display a notification of the urgency and a location of where thesurgical instrument is needed. Thus, the flag can also be based onwhether an instrument is missing from a group of surgical instruments ina different location. The computing device can also be configured topresent the group and a location associated with the group. The flag canalso identify that the encountered surgical instrument is a lostinstrument and display ownership of the instrument (i.e., return tomanufacturer).

A flag can also be based on a record for the first surgical instrumentnot being present in the datastore. The record the surgical instrumentin the datastore is updated from a plurality of secondary sources asshown in FIG. 18.

For example, FIG. 18 shows a global manufacturing database (GMDB) whichis a system that aims to incorporate a comprehensive database ofmanufacturer details for all known relevant instrument, devices, andequipment. This allows users with access to the system to be able toaccess any detail concerning an instrument or piece of equipment fromtechnical specifications to detailed instructions for use. Thiscomprehensive collection of data on all relevant manufacturer productsis enabled through two key processes: an ingest pathway known as theContent Ingest Engine (CIE), which incorporates multiple content ingestchannels, both manual and automated; and an extensive editorial systemwhich uses both automated and human-editor quality checks, to validate,categorize, and curate the ingested content.

Content that is processed by the CIE is collected through a variety ofpathways. One such way is through the access of online digital databasesincluding internal databases, and external third-party databases.Content from those databases are captured using an online crawler thatwill crawl them at regular intervals to capture new content and validateexisting content for accuracy and timeliness. Another method is throughphysical, printed materials. This is done using a scanning process thatincorporates a variety of other technology engines to interpret andanalyze those printed artifacts to parse out any relevant informationfor processing by the CIE.

Once ingested, the CIE analyzes and parses out all relevant data pointsto categorize them into their appropriate domains based on identifiedschemas and taxonomies known to the CIE. Tat content is then reviewedand edited by human Quality Control Editors (QCE) to ensure all datasets are properly identified and categorized. A final check is done byprinciple editors to validate all work done by both the CIE and the QCEsbefore becoming staged for release to the GMDB. Finally, once all datais properly vetted and staged, it is released to the GMDB and madeaccessible by any users of the system.

If the encountered surgical instrument is not present in the datastore(i.e., the first encounter for the surgical instrument), then thecomputing device can perform a record management operation to the recordfor the first surgical instrument based on the determination of theidentity type. The record management operation can include adding therecord to the datastore, adding details associated with the firstsurgical instrument manually, or combinations thereof.

A flag can be based on an economic status of the surgical instrument.For example, the economic status can refer to the cost of theinstrument, rarity, or the contractual status (i.e., under loan,purchased, or free).

The flag can be based on whether the surgical instrument is part of awrapped package. For example, if the surgical instrument is identifiedas being associated with the wrapped package, the computing device candetermine whether a chemical indicator is present with the surgicalinstrument using a machine vision technique as a subsequent operation.The record for the wrapped package is updated to indicate the presenceof the chemical indicator.

The flag can also be based on the technician performing the check-in,check-out, assembly, opening, or combinations thereof. For example, atechnician record/profile may have been flagged for previous retrainingas provided in FIG. 13. Thus, multiple actions from the flaggedtechnician can be met with strict scrutiny such as repeated verificationprompts through critical process steps or even every step. In at leastone embodiment, the flag can cause the computing device trackperformance data of a technician as shown in FIG. 14.

For example, in FIG. 14, interactions made by users (such astechnicians) can be monitored, tracked, and analyzed using a series ofmetrics to ascertain performance levels based on defined thresholds. Ifthose thresholds are not met by particular users, they will be providedwith new or remedial training curriculum developed for them throughtheir analyzed data, personalizing that training to that specificindividual. Examples of such metrics include checks against a user'sproductivity, their repeated need to access instrument information,urgent requests made by an operating room for specified instruments thatwere not available at the time of procedure, or number of flashsterilizations required for an improperly cleaned instrument.

The record associated with the technician can be updated continuouslyfrom a variety of sources (FIG. 13). For example, in FIG. 13,interactions within this system are being tracked to analyze performanceand accuracy based on a variety of key metrics. Those metrics areassessed through a variety of other subsystems and are feed in to anAnalytics Ingest Engine (AIE). The AIE captures data from thosesubsystems and quantifies them into a format that can be analyzed by anAnalytics Performance Engine (APE). The APE analyzes the data ingestedby the AIE to assess a variety of performance metrics at both theindividual level and wider facility and system level. Trends identifiedby the APE are then forwarded to the Training Development Engine (TDE)to develop training curriculum necessary for individuals who needadditional or remedial training. The APE also sends that data to anyuser-identified dashboards so that managers and directors can reviewthose data sets and trends for reporting and analytics purposes.

Training curriculum developed by the TDE are assigned to those users inneed through a multi-channel system that incorporates a variety of modesand mediums, such as immersive AR or VR environments, desktop onlinetraining, or more personalized daily activities through a user'spersonal mobile device.

Each mode will have specific modules that maximizes the learningopportunity of those modes and weighs each mode accordingly to ensurethat learning is maximized through each mode individually, but also thatlack of access to a particular mode does not inhibit a user's ability tocomplete the curriculum developed.

Completion of the curriculum is analyzed by the APE to reassess the userin question and those data points are then returned to the TDE tofurther evaluate any need for additional training.

In at least one embodiment, a profile can be developed for the userbased on the record. The record associated with the technician alsoincludes many information requests related to one or more surgicalinstruments as shown in FIG. 15. The record associated with thetechnician also includes a number of urgent requests from an operatingroom as shown in FIG. 16, meaning that wrapped packages missingcomponents (instruments or indicators) can be reflected on thetechnician's profile. The technician profile can be accessed using amanagement tool (shown in FIG. 17) that tracks performance metrics(e.g., the productivity, schedule adherence, assembly performance, andtray throughput) of an individual technician.

In block 370, the computing device can perform a record managementoperation which may include adding the surgical instrument to a group(e.g., when assembling pack) or removing the surgical instrument fromthe group (e.g., when disassembling a pack).

In block 380, the computing device can perform at least one operation inresponse to whether the surgical instrument is flagged. In addition tothe operations described herein, the computing device can start trackinginstrument time on the background surface, display cleaninginstructions, send an alert to a second computing device (such as amanager smart phone) or from an operating room computer to a centralsterile and supply department computer. In addition, the operation caninclude various personnel actions for the technician.

FIG. 4 illustrates a method 400 of using sterilization status to triggera flag, according to various embodiments.

In block 420, a technician can assemble load assist device (e.g., tray)and the computing device can track the assembly of the load assistdevice. Block 420 can correspond to method 700 in FIG. 7.

For example, in FIG. 7, uniquely identifiable trays with their uniquelyidentifiable instruments mapped accordingly in a system's LMDB and TMD,a user can be monitored for speed and accuracy at each step. Thresholdsfor speed and quality can be measured against and users who fail to meetthose thresholds will be given personalized training through analysis ofthese data points through the AIE and TDE.

Trays can be assembled by properly identifying of those uniquelyidentifiable instruments to ensure trays are equipped with the requiredinstruments. This is done through capturing an image of the instruments,which are then processed using an IRE. Properly identified instrumentsare then matched against the known tray to ensure they are proper forthe tray. This process is repeated until all instruments required forthe tray have been identified and assembled. Once assembled, the usermust then scan the required chemical indicators (CI) and biologicalindicators (BI) needed to properly process the tray during theSterilization phase. These steps are tracked in a Sterilization WorkflowEngine (SWE) and Traceability Engine (TE), which are used to monitor thesterilization process and trace instruments and trays through thatprocess. Those data points are their performance metrics are alsoforwarded for analysis by the AIE.

The computing device can determine, based on a machine vision techniquefrom the image capture device, that a wrapped package is assembled witha group of one or more surgical instruments. The computing device canalso determine whether a chemical indicator is present with the groupusing a machine vision technique. Exemplary machine vision techniquesare discussed herein.

In block 430, the wrapped package can be opened to reveal contents. Inat least one embodiment, the computing device can determine whether thewrapped package of the group is opened using the image capture device asdiscussed in copending application titled “Visual system for StatusDetection of Wrapped Packages”, which is incorporated by reference inits entirety.

In block 440, the computing device can receive a sterilization statusfor the group from the analytical device. In at least one embodiment,the analytical device can be an image capture device that is configuredto determine a status of the indicator from an image. The analyticaldevice can also be an ATP-based reader such as available under the tradedesignation Clean-Trace or a biological based reader such as availableunder the trade designation ATTEST.

In block 450, the computing device can perform at least one operationbased on the sterilization status.

For example, in block 470, if the analytical device indicate sterility,then the record for group can indicate that the wrapped package wassterile and update each member of the group accordingly. In at least oneembodiment, the computing device can update the record for the wrappedpackage to indicate the presence of the chemical indicator (independentof sterility).

In block 460, the computing device can associate a flag with a record ina datastore for the one or more surgical instruments in the group basedon the sterilization status. This can indicate that each surgicalinstrument in the group was not sterilize and trigger additionaloperations (such as user alerts or lockdown procedures).

FIG. 8 illustrates a method 800 of interacting with a template,according to various embodiments.

In block 810, the computing device can receive a template for aplurality of surgical instruments, the template comprises an imagedescribing position information for the plurality of surgicalinstruments. A template can be an orientation template that describesthe position of each instrument relative to other instruments. Thetemplate can be based on common templates or wrapped package set upsfound in the central sterile and supply department of a hospital. In atleast one embodiment, the template is retrieved from a manufacturer.Custom templates can also be created by technicians and uploaded to thedatastore.

In at least one embodiment, the template is fetched from the datastoreand related to a group of one or more instruments. A wrapped package orload assist device can have an optically active article associated withthe group of surgical instruments. In at least one embodiment, thearticle message can be read to trigger the retrieval of the template.For example, the computing device can receive the template based on datarelating to the optically active article. In at least one embodiment,the optically active article is a retroreflective.

In block 820, the computing device can receive, from the image capturedevice, a video feed of the surgical instrument. In at least oneembodiment, the surgical instrument is placed on a relatively planarsurface by a technician such that it is visible to an image capturedevice. The planar surface can also include a background surface such asa scaled mat.

In block 830, the computing device can determine a type and position ofthe surgical instrument from the video feed as discussed using themachine vision techniques described herein. The video feed can alsoinclude a chemical indicator.

In block 840, the computing device can determine whether the type andposition of the surgical instrument corresponds to a surgical instrumentof the template. This can be based on the position of the surgicalinstrument relative to an expected location of the template. Forexample, FIG. 9 illustrates various locations where instruments arematched with the position in the template FIG. 9 also illustrates thatthe tapered starter sizer awl is not in the correct position. In atleast one embodiment, the positional information for a chemicalindicator can be determined as well as shown on FIG. 12.

In response to the determination of the instrument type and position,the computing device can perform at least one operation. In block 850,the computing device can perform a display operation. In at least oneembodiment, a display operation involves modification to the display oruser interface.

For example, the computing device can display both the video feed of thesurgical instrument and the template, the surgical instrument issuperimposed over the template. The displaying can occur in real-time toallow a technician to adjust the position manually. In at least oneembodiment, the computing device can change a color of a portion of thetemplate corresponding to the surgical instrument based on a type andposition of the surgical instrument relative to the template. Forexample, as shown in FIG. 10, as a surgical instrument is moved intoposition, the template region corresponding to the position of theinstrument changes from red to green.

The computing device can also provide a zoomed video feed when thesurgical instrument is proximate to an area defined by the template forthe surgical instrument type as shown in FIG. 10. The zoomed video canaccess different functions of the image capture device (such as anoptical zoom) In at least one embodiment, the zoomed video can beprovided as a result of interaction with the user interface of thetemplate (such as an icon to zoom).

The computing device can change views of the image capture device. Forexample, changing views can include various filters or even changinglighting conditions of the video feed. For example, the computing devicecan activate a light source (such as a UV light source) if an instrumentis not in position. FIG. 11 shows an embodiment of the computing deviceactivating a UV light source for an inspection mode to detect soil.

In block 860, the computing device can conduct an inventory operation.An inventory operation can include updating the record to reflect thatthe instrument is the right type and present but in the wrong position.Alternatively, if the instrument is not present, then the record can beupdated to indicate that there is no presence of the instrument.

Various examples and implementations will be described in detail. Theseexamples should not be construed as limiting the scope of the presentdisclosure in any manner, and changes and modifications may be madewithout departing from the spirit and scope of the disclosure. Further,only some end uses have been discussed herein, but end uses notspecifically described herein are included within the scope of thepresent disclosure. As such, the scope of the present disclosure shouldbe determined only by the claims.

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored on or transmitted over, as oneor more instructions or code, a computer-readable medium and executed bya hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media, or communication media including any mediumthat facilitates transfer of a computer program from one place toanother, e.g., according to a communication protocol. In this manner,computer-readable media generally may correspond to (1) tangiblecomputer-readable storage media, which is non-transitory or (2) acommunication medium such as a signal or carrier wave. Data storagemedia may be any available media that can be accessed by one or morecomputers or one or more processors to retrieve instructions, codeand/or data structures for implementation of the techniques described inthis disclosure. A computer program product may include acomputer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, or any other medium that can be used to store desired programcode in the form of instructions or data structures and that can beaccessed by a computer. Also, any connection is properly termed acomputer-readable medium. For example, if instructions are transmittedfrom a website, server, or other remote source using a coaxial cable,fiber optic cable, twisted pair, digital subscriber line (DSL), orwireless technologies such as infrared, radio, and microwave, then thecoaxial cable, fiber optic cable, twisted pair, DSL, or wirelesstechnologies such as infrared, radio, and microwave are included in thedefinition of medium. It should be understood, however, thatcomputer-readable storage media and data storage media do not includeconnections, carrier waves, signals, or other transient media, but areinstead directed to non-transient, tangible storage media. Disk anddisc, as used, includes compact disc (CD), laser disc, optical disc,digital versatile disc (DVD), floppy disk and Blu-ray disc, where disksusually reproduce data magnetically, while discs reproduce dataoptically with lasers.

Combinations of the above should also be included within the scope ofcomputer-readable media.

Instructions may be executed by one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablelogic arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor”, as used may refer to anyof the foregoing structure or any other structure suitable forimplementation of the techniques described. In addition, in somerespects, the functionality described may be provided within dedicatedhardware and/or software modules. Also, the techniques could be fullyimplemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, an integratedcircuit (IC) or a set of ICs (e.g., a chip set). Various components,modules, or units are described in this disclosure to emphasizefunctional aspects of devices configured to perform the disclosedtechniques, but do not necessarily require realization by differenthardware units. Rather, as described above, various units may becombined in a hardware unit or provided by a collection ofinteroperative hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

It is to be recognized that depending on the example, certain acts orevents of any of the methods described herein can be performed in adifferent sequence, may be added, merged, or left out altogether (e.g.,not all described acts or events are necessary for the practice of themethod). Moreover, in certain examples, acts or events may be performedconcurrently, e.g., through multithreaded processing, interruptprocessing, or multiple processors, rather than sequentially.

In some examples, a computer-readable storage medium includes anon-transitory medium. The term “non-transitory” indicates, in someexamples, that the storage medium is not embodied in a carrier wave or apropagated signal. In certain examples, a non-transitory storage mediumstores data that can, over time, change (e.g., in RAM or cache).

Those having skill in the art will appreciate that many changes may bemade to the details of the above-described examples and implementationswithout departing from the underlying principles thereof. The scope ofthe present disclosure should, therefore, be determined only by thefollowing claims.

In some examples, an apparatus (e.g., an image capture device) orcomputing device comprises least one single core or multi core computerprocessing unit (CPU) and/or graphics processing unit (GPU). In someexamples, the CPU is co-located with a camera, that is, disposed withinproximity to the camera. In some examples, the CPU is mounted on thesame board as the camera. In other examples, the CPU is not co-locatedwith the camera and is connected to the camera by other means ofcommunication, such as, for example, coaxial cables and/or wirelessconnections. In some examples, the CPU substantially concurrentlyprocesses multiple frames via operating system provided services, suchas, for example, time slicing and scheduling. In other examples, theapparatus further comprises at least one multi-core CPU.

In some examples an apparatus or computing device produces bundles ofdata including, for example, date, time, images, barcode read data,Optical Character Recognition (OCR) read data, and other metadata, thatmay be useful in sterilization.

In some examples, pre-processing may increase the rate of processingimages. In some examples, intelligent selection is performed viafield-programmable gate array (FPGA) pre-processing which can processmultiple channels at 50 fps. As an example, fifteen images may beprocessed by OCR from a first channel, but only three barcode imagesfrom a second channel may be processed during the same period. Thisdifference in the number of images processed per channel may happen whenone of the images (e.g., barcode image) is more complex.

The radiation detected by the camera can come from any of a number ofsources. Of interest is the radiation reflected from the opticallyactive article, and specifically, the amount of radiation reflected fromeach area inside that region of interest on the article. The camera ordetection system collects radiation from each region of the opticallyactive article with the goal of creating a difference (contrast) betweenthe background and/or between each indicia or piece of identifyinginformation on the optically active article. Contrast can be affected innumerous ways, including the use of coaxial radiation to overwhelm theamount of ambient radiation. The use of filters on the camera can helpaccentuate the differences between the indicia or identifyinginformation and background by selectively removing undesired radiationwavelengths and passing only the desired radiation wavelengths.

LIST OF ILLUSTRATIVE EMBODIMENTS

1. A system comprising:

an image capture device;

a computing device communicatively coupled to the image capture device,the computing device comprising one or more computer processors and amemory comprising instructions that when executed by the one or morecomputer processors cause the one or more computer processors to:

receive a test image from the image capture device corresponding to afirst surgical instrument;

determine an identity type of the first surgical instrument using thetest image in a machine vision technique;

determining whether the first surgical instrument is flagged;

perform at least one operation in response to whether the first surgicalinstrument is flagged.

2. The system of embodiment 1, wherein the computing device comprises amemory comprising instructions that when executed by the one or morecomputer processors cause the one or more computer processors to:

adjust a confidence of the determination of the identity type based onan identity parameter.

3. The system of any of embodiments 1 or 2, further comprising:

a datastore comprising one or more records for the first surgicalinstrument.

4. The system of embodiment 3, wherein the flag is derived from the oneor more records from the datastore.5. The system of any of embodiments 1 to 4, further comprising:

a display communicatively coupled to the computing device.

6. The system of any of embodiments 4 to 5, wherein the first surgicalinstrument is flagged based on an economic status of the first surgicalinstrument,

wherein the computing device comprises a memory comprising instructionsthat when executed by the one or more computer processors cause the oneor more computer processors to:

present, via the display, a visual indication of the economic status ofthe first surgical instrument.

7. The system of any of embodiments 1 to 3, wherein the first surgicalinstrument is flagged based on special handling instructions for thefirst surgical instrument;

wherein to perform at least one operation the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to present the specialhandling instructions on the display.

8. The system of any of embodiments 1 to 7, wherein the first surgicalinstrument is flagged based on whether the first surgical instrument ismissing from a group;

wherein to perform at least one operation the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to present the group and alocation associated with the group.

8a. The system of any of embodiments 1 to 7, wherein the first surgicalinstrument is flagged based on whether the first surgical instrument isbroken or damaged;

wherein to perform at least one operation the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to present an alert via a userinterface.

8b. The system of any of embodiments 1 to 7, wherein the first surgicalinstrument is flagged based on whether an estimated processing time ofthe first surgical instrument is within a time threshold of an operatingroom schedule;

wherein to perform at least one operation the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to present an alert via a userinterface.

8c. The system of embodiment 8b, wherein the estimated processing timeis based on real-time and historical device reprocessing data that iscaptured and analyzed by a machine vision technique.9. The system of any of embodiments 1 to 8, wherein the first surgicalinstrument is flagged based on a record for the first surgicalinstrument not being present in the datastore;

wherein to perform at least one operation the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to perform a record managementoperation to the record for the first surgical instrument based on thedetermination of the identity type.

10. The system of embodiment 9, wherein the record management operationcomprises adding the record to the datastore.11. The system of embodiment 10, wherein the record management operationcomprises adding details associated with the first surgical instrumentmanually.12. The system of embodiment 11, wherein to determine the identity typeof the first surgical instrument, the memory comprises instructions thatwhen executed by the one or more computer processors cause the one ormore computer processors to:

determine, with the image capture device, whether an article message ispresent on a surgical instrument; and

determine the identity type of the first surgical instrument using amachine vision technique based on whether the article message ispresent.

13. The system of embodiment 12, wherein the article message is a barcode, QR code, visual indication of an alphanumeric sequence, orcombinations thereof.14. The system of embodiment 13, wherein the bar code comprises colorcoded bar code stripes.15. The system of embodiment 11 or 12, wherein to determine whether thearticle message is present occurs responsive to determining if a receiptor purchase order is available for the device; and

if available, adding the receipt or purchase order to the record.

16. The system of any of embodiments 1 to 15, wherein whether theinstrument is part of a wrapped package results in a flag;

wherein to perform at least one operation the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to determine whether achemical indicator is present with the first surgical instrument using amachine vision technique;

wherein the record for the wrapped package is updated to indicate thepresence of the chemical indicator.

17. The system of any of embodiments 1 to 16, wherein to perform atleast one operation the memory comprises instructions that when executedby the one or more computer processors cause the one or more computerprocessors to track performance data of a technician.18. The system of any of embodiments 1 to 17, wherein to perform atleast one operation the memory comprises instructions that when executedby the one or more computer processors cause the one or more computerprocessors to update a record associated with a technician.19. The system of embodiment 18, wherein the record associated with thetechnician also includes a number of information requests related to oneor more surgical instruments.20. The system of embodiment 19, wherein the record associated with thetechnician also includes a number of urgent requests from an operatingroom.21. The system of any of embodiments 1 to 20, wherein a record for thefirst surgical instrument in the datastore is updated from a pluralityof secondary sources.22. The system of any of embodiments 1 to 21, wherein to determine theidentity type, the memory comprises instructions that when executed bythe one or more computer processors cause the one or more computerprocessors to:

receive a plurality of images corresponding to a plurality of surgicalinstruments comprising a first set of images that include surgicalinstruments having the instrument type;

generate a first set of feature values based on a visual representationof a surgical instrument in the image, wherein the first set of featurevalues correspond to features usable to determine an instrument type ofthe surgical instrument;

-   -   apply the first set of feature values for the first set of        images to a first model to determine affiliation with a        plurality of clusters based on shared features, wherein the        first model does not use labels of the set of images;    -   assign an image from the first set of images to a cluster.        23. The system of embodiment 22, wherein one or more processors        are configured to perform at least one operation based at least        in part on the cluster assigned.        24. The system of embodiment 22, wherein one or more processors        are configured to:

receive a second set of images corresponding to a first surgicalinstrument;

generate a second set of feature values from the second set of images;

apply the second set of feature values to a first plurality of models todetermine the instrument type, each of the first plurality of modelscorresponds to a cluster from the plurality of clusters;

generate, based at least in part on the one or more processorsprocessing the set of feature values with at least one model, anindication of the instrument type for the first surgical instrument; and

perform at least one operation based at least in part on the instrumenttype.

25. The system of embodiment 22, wherein at least one model from thefirst plurality of models is a supervised learning model executed by theone or more processors and the second set of feature values are inputinto the neural network.26. The system of any of embodiments 20 to 25, wherein the memorycomprises instructions that when executed by the one or more computerprocessors cause the one or more computer processors to:

-   -   apply a third set of feature values for a portion of the first        set of images to a second model to determine affiliation with a        plurality of subclusters for a cluster based on shared features,        wherein the second model does not use labels of the set of        images;    -   apply the second set of feature values to a second plurality of        models to determine the instrument type, each of the second        plurality of models corresponds to a subcluster from the        plurality of subclusters.        27. The system of any of embodiments 20 to 26, wherein the        memory comprises instructions that when executed by the one or        more computer processors cause the one or more computer        processors to:        generate the indication of the instrument type based on ensemble        modeling an output of each model of the plurality of models.        28. The system of any of embodiments 20 to 27, wherein the        memory comprises instructions that when executed by the one or        mon computer processors cause the one or more computer        processors to:        transfer characteristics related to the second set of images        between models of the first plurality of models and the first        model.        29. The system of any of embodiments 20 to 28, wherein the first        model is an unsupervised learning model.        30. The system of any of embodiments 20 to 29, wherein the first        set of images is different than the second set of images.        31. The system of any of embodiments 20 to 30, wherein the        neural network is a convolutional neural network.        32. The system of any of embodiments 20 to 31, wherein the        unsupervised learning model implements a gaussian mixture model.        33. The system of any of embodiments 20 to 32, wherein the        supervised learning model uses a Bayesian model.        34. The system of any of embodiments 20 to 33, wherein the        supervised learning model uses at least one of logistic        regression, random forest, support vector machine, or boosting.        35. The system of any of embodiments 20 to 34, wherein the first        surgical instrument is disposed on a background surface;

wherein the memory comprises instructions that when executed by the oneor more computer processors cause the one or more computer processors toreceive the plurality of images from the image capture device, andpre-process an image of the plurality of images.

36. The system of embodiment 35, wherein, to pre-process the image, thememory comprises instructions that when executed by the one or morecomputer processors cause the one or more computer processors to:

receive an image of the first surgical instrument and the backgroundsurface from the image capture device;

generate a mask from the image by removing the background surface;

remove noise from the mask;

apply the mask to the image.

37. The system of embodiment 36, wherein to apply the mask, the memorycomprises instructions that when executed by the one or more computerprocessors cause the one or more computer processors to align the maskwith the feature.38. The system of any of embodiments 35 to 37, further comprising:a group of first surgical instruments wherein the background surface isan unwrapped sterilization pack;a chemical indicator proximate to the group of first surgicalinstruments;wherein at least one of the features generated by the one or moreprocessors is an indicator status of the chemical indicator and thesecond set of images comprises the chemical indicator.39. The system of any of embodiments 36 to 38, wherein to generate amask, the memory compnses instructions that when executed by the one ormore computer processors cause the one or more computer processors to:

apply a background subtraction operation to the image;

modify one or more color channels of the image;

apply a morphological operator.

40. The system of any of embodiments 36 to 39, wherein to remove noisefrom mask, the memory comprises instructions that when executed by theone or more computer processors cause the one or more computerprocessors to:

-   -   extract a plurality of connected components from the image;    -   determine a first group of connected components that are smaller        than a first threshold dimension;    -   determine a second group of connected components that are larger        than a second threshold dimension;    -   remove a connected component from the first and second group        that is within a threshold distance from an edge of the image.        41. The system of any of embodiments 22 to 40, wherein to        receive the image corresponding to the first surgical        instrument, the memory comprises instructions that when executed        by the one or more computer processors cause the one or more        computer processors to:        receive a masked image for the first surgical instrument from a        datastore.        42. The system of any of embodiments 22 to 41, wherein to        provide the plurality of images corresponding to the first        surgical instrument, the memory comprises instructions that when        executed by the one or more computer processors cause the one or        more computer processors to:        select the first surgical instrument based on an instrument        attribute.        43. A system comprising:

an image capture device;

an analytical device;

a computing device communicatively coupled to the image capture deviceand the analytical device, the computing device comprising one or morecomputer processors and a memory comprising instructions that whenexecuted by the one or more computer processors cause the one or morecomputer processors to:

determine, based on a machine vision technique from the image capturedevice, that a wrapped package is assembled with a group of one or moresurgical instruments;

receive a sterilization status for the group from the analytical device;

-   -   perform at least one operation based on the sterilization        status.        44. The system of embodiment 43, wherein to perform at least one        operation the memory comprises instructions that when executed        by the one or more computer processors cause the one or more        computer processors to flag a record in a datastore for the one        or more surgical instruments in the group based on the        sterilization status.        45. The system of embodiment 43 or 44, wherein the memory        comprises instructions that when executed by the one or more        computer processors cause the one or more computer processors to        determine whether the wrapped package of the group is opened        using the image capture device.        46. The system of any of embodiments 43 to 45, wherein, to        determine that a wrapped package is assembled, the memory        comprises instructions that when executed by the one or more        computer processors cause the one or more computer processors to        determine whether a chemical indicator is present with the group        using a machine vision technique;

wherein performing the at least one operation comprises updating therecord for the wrapped package to indicate the presence of the chemicalindicator.

47. The system of any of embodiments 43 to 46, wherein to determine thata wrapped package is assembled, the memory comprises instructions thatwhen executed by the one or more computer processors cause the one ormore computer processors to determine whether a surgical instrument anda chemical indicator are present.48. The system of embodiment 47, wherein to determine whether a surgicalinstrument and chemical indicator are present, the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to:

receive an image corresponding to a surgical instrument and a chemicalindicator;

generate a set of feature values based on a visual representation of thefirst surgical instrument in the image, wherein the set of featurevalues correspond to features usable to determine an instrument type ofthe surgical instrument, wherein at least one of the set of featurevalues is the chemical indicator;

apply the set of feature values to at least one model that is trainedbased at least in part on a set of images that include surgicalinstruments having the instrument type, to determine the instrument typeand the chemical indicator; and

performing at least one operation based at least in part on theinstrument type and the chemical indicator.

49. The system of embodiment 48, wherein the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to:

pre-process the image responsive to receiving the image.

50. The system of embodiment 49, further comprising a backgroundsurface.51. The system of any of embodiments 48 or 50, wherein to pre-processthe image, the memory comprises instructions that when executed by theone or more computer processors cause the one or more computerprocessors to:

receive the image of the instrument and a background surface from theimage capture device;

generate a mask from the image by removing the background surface;

remove noise from the mask;

apply the mask to the image.

52. The system of embodiment 51, wherein to apply the mask, the memorycomprises instructions that when executed by the one or more computerprocessors cause the one or more computer processors to align the maskwith the feature.53. The system of embodiment 51, wherein to generate a mask, the memorycomprises instructions that when executed by the one or more computerprocessors cause the one or more computer processors to:

apply a background subtraction to the image;

modify color channels of the image;

apply a morphological operator.

54. The system of embodiment 51, wherein to remove noise from mask thememory comprises instructions that when executed by the one or moncomputer processors cause the one or more computer processors to:extract a plurality of connected components from the image;determine a first group of connected components that are smaller than afirst threshold dimension;determine a second group of connected components that are larger than asecond threshold dimension;remove a connected component from the first and second group that iswithin a threshold distance from an edge of the image.55. The system of embodiment 54, wherein applying the set of featurevalues to at least one model comprises:

identifying an instrument feature of the surgical instrument relative toa plurality of instrument features from a plurality of instruments;

determining an instrument type based on a comparison with the pluralityof instrument features for a subset of the plurality of instruments.

56. The system of embodiment 55, wherein the instrument featurecomprises at least one of the length, width, length/width ratio, area,perimeter, histogram of oriented gradients feature, or Gabor-likefeatures of a connected component.57. The system of any of embodiments 47 to 56, wherein the at least onemodel includes a supervised machine learning model.58. The system of any of embodiments 47 to 57, wherein to determine aninstrument type, the memory comprises instructions that when executed bythe one or more computer processors cause the one or more computerprocessors to:

receive a test image of a plurality of instruments and the backgroundsurface from the image capture device;

apply the mask for a first surgical instrument to the test image;

apply a pattern matching model to an image for the first surgicalinstrument relative and the masked test image;

determine that the masked test image is matches the instrument type asthe first surgical instrument based on the pattern matching model.

59. The system of embodiment 58, wherein to apply a pattern matchingmodel, the memory comprises instructions that when executed by the oneor more computer processors cause the one or more computer processorsto:

extract keypoints from the masked test image;

determine the keypoints that match between the masked test image and animage for the first surgical instrument;

determine that the first surgical instrument is present based on aquantity of matched keypoints.

60. The system of embodiment 58, wherein the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to determine that the firstsurgical instrument is present based on a score associated with matchingpairs of keypoints.61. A system comprising:

an image capture device;

a surgical instrument.

a display;

a computing device communicatively coupled to the image capture device,the computing device comprising one or more computer processors and amemory comprising instructions that when executed by the one or morecomputer processors cause the one or more computer processors to:

receive a template for a plurality of surgical instruments, the templatecomprises an image describing position information for the plurality ofsurgical instruments:

receive, from the image capture device, a video feed of the surgicalinstrument,

determine a type and position of the surgical instrument from the videofeed;

determine whether the type and position of the surgical instrumentcorresponds to a surgical instrument of the template;

perform at least one operation in response to the determination of theidentity of the one or more surgical instruments.

62. The system of embodiment 61, wherein the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to:

display both the video feed of the surgical instrument and the template,the surgical instrument is superimposed over the template.

63. The system of embodiment 62, wherein the at least one operationcomprises providing a zoomed video feed when the surgical instrument isproximate to an area defined by the template for the surgical instrumenttype.64. The system of embodiment 61 or 62, wherein the at least oneoperation comprises changing a color of a portion of the templatecorresponding to the surgical instrument based on a type and position ofthe surgical instrument relative to the template.65. The system of any of embodiments 61 to 63 wherein receiving thetemplate for a plurality of surgical instruments comprises reading anoptically active article on the package;

receiving the template based on data relating to the optically activearticle.

66. The system of embodiment 65, wherein the optically active article isa retroreflective.67. The system of embodiment 61, further comprising a planar surface,wherein the surgical instrument is placed on the planar surface68. The system of any of embodiments 61 to 67, wherein the at least oneoperation is modifying a record for the first surgical instrument on adatastore based at least in part on the instrument type of the firstsurgical instrument.68a. The system of any of embodiments 61 to 67, wherein the memorycomprises instructions that when executed by the one or more computerprocessors cause the one or more computer processors to: dispensepre-cleaning solution by evaluated color of a chemical solution todetermine proper dilution level for the first surgical instrument.69. The system of embodiments 1 to 54 or system of embodiments 55 to 68,comprising a datastore, wherein the modifying the record comprises atleast one of adding a record, changing a record, or deleting a record inresponse to determining the instrument type.70. The system of embodiment 69, wherein adding a record occursresponsive to the first surgical instrument not being present on thedatastore.71. The system of embodiment 70, wherein the first surgical instrumentis not present during an instrument check in process.72. The system of embodiment 69, wherein the one or more processors isconfigured to associate at least one image of the first surgicalinstrument with the record.73. The system of any of embodiments 72, wherein the one or moreprocessors is configured to associate the record of the first surgicalinstrument with a plurality of instruments in a group.74. The system of any of embodiments 69 to 73, wherein the datastorecomprises technician cleaning instructions associated with a record forthe first surgical instrument.75. The system of embodiment 74, wherein the technician cleaninginstructions comprises a video.76. The system of embodiment 74 or 75, wherein the datastore comprises aplurality of time milestones based on the technician cleaninginstructions associated with the record for the first surgicalinstrument.77. The system of any of embodiments 69 to 76, further comprising adisplay, wherein the display is configured to display at least some dataassociated with the record.78. The system of embodiment 76, wherein the one or more processors areconfigured to capture a video feed from the image capture device anddetermine whether at least one of the plurality of time milestones ismet.79. The system of any of embodiments 69 to 78, wherein the operation ofthe one or more processors comprises accessing the record from thedatastore and determine based on the image received from the imagecapture device whether a second surgical instrument is present.80. The system of embodiment 79, wherein the operation is to alert auser if the second surgical instrument is not present.81. The system of any of embodiments 69 to 76, further comprising aquantitative biological tester that provides an indication of biologicalactivity of a surface.82. The system of embodiment 81, wherein the quantitative biologicaltester determines the indication from the first instrument and providesthe indication to the datastore so that the indication is associatedwith the record.83. The system of any of embodiments 69 to 82, wherein the one or moreprocessors are configured to determine a prioritization for a group ofinstruments based on the record.84. The system of any of embodiments 69 to 83, wherein the one or moreprocessors are configured to determine a group of surgical instrumentsin a pack from a plurality of records, and determining the surgicalinstruments not present based on the based on the second set of images.85. A computer-implemented method comprising:

receive a test image, from the image capture device, corresponding to afirst surgical instrument,

determine an identity type of the first surgical instrument using thetest image in a machine vision technique;

determining whether the first surgical instrument is flagged;

perform at least one operation in response to whether the first surgicalinstrument is flagged.

86. The method of embodiment 85, further comprising:

adjusting a confidence of the determination of the identity type basedon an identity parameter

87. The method of embodiment 85 or 86, wherein the flag is derived fromthe one or more records from the datastore.88. The method of any of embodiments 85 to 87, further comprising:

presenting, via a display, a visual indication of the economic status ofthe first surgical instrument:

wherein the first surgical instrument is flagged based on an economicstatus of the first surgical instrument.

89. The method of any of embodiments 85 to 88, wherein the firstsurgical instrument is flagged based on special handling instructionsfor the first surgical instrument:

wherein performing at least one operation comprises presenting thespecial handling instructions on the display.

90. The method of any of embodiments 85 to 89, wherein the firstsurgical instrument is flagged based on whether the first surgicalinstrument is missing from a group:

wherein performing at least one operation comprises presenting the groupand a location associated with the group.

91. The method of any of embodiments 85 to 90, wherein the firstsurgical instrument is flagged based on a record for the first surgicalinstrument not being present in the datastore

wherein performing at least one operation comprises performing a recordmanagement operation to the record for the first surgical instrumentbased on the determination of the identity type.

92. The method of embodiment 91, wherein the record management operationcomprises adding the record to the datastore.93. The method of embodiment 92, wherein the record management operationcomprises adding details associated with the first surgical instrumentmanually.94. The method of embodiment 93, wherein determining the identity typeof the first surgical instrument comprises:

determining, with the image capture device, whether an article messageis present on a surgical instrument; and

determining the identity type of the first surgical instrument using amachine vision technique based on whether the article message ispresent.

95. The method of embodiment 94, wherein the article message is a barcode, QR code, visual indication of an alphanumeric sequence, orcombinations thereof.96. The method of embodiment 95, wherein the bar code comprises colorcoded bar code stripes.97. The method of any of embodiments 94 to 96, wherein determiningwhether the article message is present occurs responsive to determiningif a receipt or purchase order is available for the device; and

if available, adding the receipt or purchase order to the record.

98. A computer-implemented method, comprising:

determining, based on a machine vision technique from the image capturedevice, that a wrapped package is assembled with a group of one or moresurgical instruments,

receiving a sterilization status for the group from an analyticaldevice,

performing at least one operation based on the sterilization status.

99. The method of embodiment 98, wherein performing at least oneoperation comprises flagging a record in a datastore for the one or moresurgical instruments in the group based on the sterilization status.100. A computer implemented method comprising:

receiving a template for a plurality of surgical instruments, thetemplate comprises an image describing position information for theplurality of surgical instruments;

receiving, from the image capture device, a video feed of the surgicalinstrument;

determining a type and position of the surgical instrument from thevideo feed;

determining whether the type and position of the surgical instrumentcorresponds to a surgical instrument of the template;

performing at least one operation in response to the determination ofthe identity of the one or more surgical instruments.

101. The method of embodiment 100, further comprising: displaying boththe video feed of the surgical instrument and the template, the surgicalinstrument is superimposed over the template.102. The method of embodiment 100, wherein performing the at least oneoperation comprises providing a zoomed video feed when the surgicalinstrument is proximate to an area defined by the template for thesurgical instrument type.

What is claimed is:
 1. A system comprising: an image capture device; acomputing device communicatively coupled to the image capture device,the computing device comprising one or more computer processors and amemory comprising instructions that when executed by the one or morecomputer processors cause the one or more computer processors to:receive a test image from the image capture device corresponding to afirst surgical instrument; determine an identity type of the firstsurgical instrument using the test image in a machine vision technique;determine whether the first surgical instrument is flagged; perform atleast one operation in response to whether the first surgical instrumentis flagged.
 2. The system of claim 1, wherein the computing devicecomprises a memory comprising instructions that when executed by the oneor more computer processors cause the one or more computer processorsto: adjust a confidence of the determination of the identity type basedon an identity parameter.
 3. The system of claim 1, further comprising:a datastore comprising one or more records for the first surgicalinstrument.
 4. The system of claim 3, wherein a flag is derived from theone or more records from the datastore.
 5. The system of claim 1,further comprising: a display communicatively coupled to the computingdevice.
 6. The system of claim 5, wherein the first surgical instrumentis flagged based on an economic status of the first surgical instrument,wherein the computing device comprises a memory comprising instructionsthat when executed by the one or more computer processors cause the oneor more computer processors to: present, via the display, a visualindication of the economic status of the first surgical instrument. 7.The system of claim 6, wherein the first surgical instrument is flaggedbased on special handling instructions for the first surgicalinstrument; wherein to perform at least one operation the memorycomprises instructions that when executed by the one or more computerprocessors cause the one or more computer processors to present thespecial handling instructions on the display.
 8. The system of claim 7,wherein the first surgical instrument is flagged based on whether thefirst surgical instrument is missing from a group; wherein to perform atleast one operation the memory comprises instructions that when executedby the one or more computer processors cause the one or more computerprocessors to present the group and a location associated with thegroup.
 9. The system of claim 8, wherein the first surgical instrumentis flagged based on a record for the first surgical instrument not beingpresent in the datastore; wherein to perform at least one operation thememory comprises instructions that when executed by the one or morecomputer processors cause the one or more computer processors to performa record management operation to the record for the first surgicalinstrument based on the determination of the identity type.
 10. Thesystem of claim 9, wherein the record management operation comprisesadding the record to the datastore.
 11. The system of claim 10, whereinthe record management operation comprises adding details associated withthe first surgical instrument manually.
 12. The system of claim 11,wherein to determine the identity type of the first surgical instrument,the memory comprises instructions that when executed by the one or morecomputer processors cause the one or more computer processors to:determine, with the image capture device, whether an article message ispresent on a surgical instrument; and determine the identity type of thefirst surgical instrument using a machine vision technique based onwhether the article message is present.
 13. The system of claim 12,wherein the article message is a bar code, QR code, visual indication ofan alphanumeric sequence, or combinations thereof.
 14. The system ofclaim 13, wherein the bar code comprises color coded bar code stripes.15. The system of claim 12, wherein to determine whether the articlemessage is present occurs responsive to determining if a receipt orpurchase order is available for the device; and if available, adding thereceipt or purchase order to the record.
 16. A system comprising: animage capture device; an analytical device; a computing devicecommunicatively coupled to the image capture device and the analyticaldevice, the computing device comprising one or more computer processorsand a memory comprising instructions that when executed by the one ormore computer processors cause the one or more computer processors to:determine, based on a machine vision technique from the image capturedevice, that a wrapped package is assembled with a group of one or moresurgical instruments; receive a sterilization status for the group fromthe analytical device; perform at least one operation based on thesterilization status.
 17. The system of claim 16, wherein to perform atleast one operation the memory comprises instructions that when executedby the one or more computer processors cause the one or more computerprocessors to flag a record in a datastore for the one or more surgicalinstruments in the group based on the sterilization status.
 18. A systemcomprising: an image capture device; a surgical instrument; a display; acomputing device communicatively coupled to the image capture device,the computing device comprising one or more computer processors and amemory comprising instructions that when executed by the one or morecomputer processors cause the one or more computer processors to:receive a template for a plurality of surgical instruments, the templatecomprises an image describing position information for the plurality ofsurgical instruments; receive, from the image capture device, a videofeed of the surgical instrument; determine an instrument type andposition of the surgical instrument from the video feed; determinewhether the type and position of the surgical instrument corresponds toa surgical instrument of the template; perform at least one operation inresponse to the determination of the instrument type of the surgicalinstrument.
 19. The system of claim 18, wherein the memory comprisesinstructions that when executed by the one or more computer processorscause the one or more computer processors to: display both the videofeed of the surgical instrument and the template, the surgicalinstrument is superimposed over the template.
 20. The system of claim18, wherein the at least one operation comprises providing a zoomedvideo feed when the surgical instrument is proximate to an area definedby the template for the instrument type.